|
Software testing is an integral and critical phase of any software project that takes up almost 40% of a software development life cycle, ideally. Developers and testers must ensure that new products or product enhancements meet functional and performance requirements, that those products are reliable and able to operate consistently under peak load. The risk of releasing a product that is not yet ready are greater now than ever before because of the change in expectations and demands of end users. According to the Standish Group's (an international research advisory firm) 2003 research report on Project Failure and Success, nearly three out of four software projects in the US are either delivered late, overbudget or are cancelled before being completed. Project success rates are just 34% of all projects while failures have declined to 15%. Challenged projects account for the remaining 51%. Agile development and testing is growing in popularity and smart QA/testing teams keep pace with current development trends. Agile testing tools vary from project management tools to automated testing tools. We recently spoke with JoEllen West, Agile tester from VersionOne, to discuss automated testing and its' role in Agile environments. VersionOne is recognized by agile practitioners as a leader in Agile project management tools and JoEllen is the lead tester on their agile development team. Automated Testing of Agile Projects vs. Waterfall Projects JoEllen had this to say about the difference: "Automation is a critical component to maintaining agility, and is a priority for the entire team through established practices/disciplines and a focus on continuous improvement. Continuous integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of applying automation beyond the scope of traditional automated tests." "In contrast, waterfall projects will tend not to invest in automation as a priority, primarily due to the phased investment and handoff mindset that accompanies that approach. At best, builds may leverage some automation, and a phase of a waterfall project may be dedicated to functional test automation for long-term support/future release needs, but it's rare to find much more in a back-end-loaded waterfall project." Agile Tester Skills Regarding agile tester skills, JoEllen said that an Agile tester will have a more team oriented approach and use a wide variety of tools. "In addition to a quality/test-oriented outlook, an agile tester will incorporate a wide array of testing tools and technologies into their testing efforts. They are an integral member of the development team, participating fully in the development process, from design through delivery & deployment. A deep understanding of the domain and features under development is essential to developing tests that provide meaningful feedback. Testers may also work on several stories or features at one time, so tools to organize and juggle multiple testing activities are critical. They will have adapted to a more collaborative, team-based approach to the iterative/continuous delivery of high-quality, working software as a measure of progress." Automation Responsibilities on an Agile Team "The team! Literally anyone who does anything repetitively more than 2 or 3 times in a short period of time should and will automate it, preferably with a well known tool such as TestComplete, open source options or homegrown scripts/utilities. Testers generally shoulder the development of functional test automation, since that activity can parallel the actual feature development." Deciding what Tests get Automated in Agile Environments "The individuals doing detailed planning for a given user story (i.e. product owner(s), developers, testers) may make some of those decisions at that time. Developers and testers working on the story make detailed decisions about where automation can be used to trigger acceptance/signal completion of the story. During testing, testers may see opportunities for adhoc, on the fly automation to increase test coverage." Maintaining and Reviewing Automated Tests in Agile Environments "Since tests are often created by tester-developer and/or tester-product owner pairs, there's a built-in review process of the test scope and validation level as part of story acceptance. Otherwise, the collaborative planning to define the tests, then the collaborative acceptance with the product owner provides ample shared knowledge and review. The end result is robust, resilient automated tests which provide the backbone of regression testing, and any test failures from regression testing may also trigger a test review." "Ideally, the entire team maintains all tests, whether unit, functional, or any other. However, testers may use specialized tools for GUI-level tests that require their involvement to maintain tests over time. To reduce the possible silo-ing of this information, the tools should be carefully chosen to be accessible to all team members, information should be readily available, and the test inventory should be continuously culled to prevent test bloat. Beware of the tester building a mammoth suite of tests in a specialized environment as a job security strategy." Agile Teams Automate Black Box and White Box Tests "For functional/acceptance tests, it's most common for there to be a combination of white-box (under-the-covers) and UI automation (black-box) tests. The white-box tests tend to be less fragile, so are less costly to maintain, whereas the black-box UI-driven tests need to be leveraged judiciously due to the cost to develop and maintain the tests over time. Ideally, testers are involved in both, so that the black-box tests can supplement instead of duplicating white-box tests." Implementing Test Driven Development "By definition, TDD is a developer coding practice (via a unit test framework), not an agile tester practice. Simply, a developer will write a failing unit test, add the associated code to satisfy the test, then re-run those tests to ensure success. Subsequent refactoring/redesign of the resulting code is done whilst keeping the unit tests running 'Green'. The standard TDD mantra is Red-Green-Refactor." "However, the same concept (i.e. test-first) is definitely applied to higher-level functional and acceptance testing, whether automated or executed manually. Essentially, detailed requirements for a user story are expressed in the form of acceptance tests, which can then be captured in advance in a test automation environment such that they will eventually run 'green' once the functionality is in place, signaling completion (they may run 'yellow' on incomplete stories)." When Not to Use Automation in an Agile Environment It's important to know when and when not to automate tests, this holds true no matter what methodology is used. JoEllen said that if a test is only performed once then it shouldn't be automated. "One-time tasks and exploratory testing/edge cases shouldn't be automated. Edge cases are, by definition, usually one-off test cases, and the effort to automate edge cases generally does not pay off. Exploratory tests are best used to gain knowledge of a new feature and then tweak or revise tests based on the new knowledge." Switching from Waterfall to Agile When asked if she had any advice to give testers who are switching from a Waterfall project to an Agile project, JoEllen had this to say: "Be aware of the change in mindset/culture. Experiment with modern tools. Keep the big picture of the application in mind and consider individual user stories within the bigger context. Think holistically, and encourage the rest of the team to do so as well." Functional, GUI, Regression and Performance Testing in an Agile Environment JoEllen explained that all testing types have a place in an Agile shop, it is just a matter of how they are used. "Unit, functional, acceptance, GUI (some of which become regression), performance and load testing are all used. We try to select the most appropriate tool for the team and the type of test we're performing, so we have several different tools in our arsenal." "All of these testing types have a place in an agile environment. However, it's important to incorporate them in an Agile way, rather than a prescriptive way. Balance the quality goals of the team with the requirements and risk of individual user stories." Automated Testing Starts on Day One The automated testing process begins the same time as development in an Agile shop. JoEllen said, "The testing process begins at the same time as development - Day One." Automated Testing Tool Features - What is More Important to Agile Testers? "Most agile teams, regardless of any specific methodology, are looking for fast, reliable feedback from their automated tests. Fast, reliable feedback means timely results with minimal false failures, ideally incorporated as part of the continuous integration (build) pipeline." Automated Testing ROI in Agile Environments "It's really a prerequisite to becoming a truly agile team. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles. Thus, if ROI is producing high-quality software iteratively, then yes, you see a nice return on your automation." Automated Testing and Agile Methodologies JoEllen explained automated testing's role in different Agile methodologies: "In XP, automation is explicitly called out in a variety of disciplines (TDD, continuous integration, automated acceptance testing). Scrum teams may have incorporated those practices on their path to agility, but automation is not prescribed by the base methodology. Lean product development assumes automation in the pursuit of waste elimination." Implementing Automated Testing in Agile Environments - Challenges JoEllen explained the challenges frequently seen in Agile shops: "Availability of people & skills, a holistic team-oriented approach, striking the right balance between automation and excess test inventory which can result in drag. Some teams struggle with when to automate functional tests � whether to delay acceptance of a user story until the automated tests are complete, or to have automated test development as a separate activity. We've found that with the right tools, development of GUI-level tests can easily take place as standard work on a user story. However, automating functional tests on a story-by-story basis may lead to test bloat and may not provide the higher-level coverage that's desired from these tests. Teams may choose to write user stories to prioritize and schedule test development at that level." Automated Testing Benefits in an Agile Environment JoEllen said that the same benefits seen in other areas of testing are also seen in Agile shops. "Speed, quality, malleable code base (i.e. adaptable/open to change), feedback, efficiency are all benefits that we experience using automation for our tests." TestComplete in Agile Environments JoEllen had nice things to say about TestComplete for Agile testing: "A co-worker's prior company chose it over the high-end commercial offerings and really digs it for automating acceptance testing with both client-server and browser-based applications. It's both developer and tester-friendly, so it reduces the risk/tendency of specialization/single-role maintainability of the test suite." About VersionOne Since 2002, companies such as Adobe, BBC, Siemens, Disney, Dow Chemical, IBM, Lockheed Martin, Sony, 3M and Business Objects have turned to VersionOne to help provide greater value to their customers. Today more than 10,000 teams and 70,000 users in 50 countries use VersionOne's Agile project management tools to streamline and standardize their Agile development efforts - See more at: http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/#sthash.qR2YBNmZ.dpufAgile development and testing is growing in popularity and smart QA/testing teams keep pace with current development trends. Agile testing tools vary from project management tools to automated testing tools. We recently spoke with JoEllen West, Agile tester from VersionOne, to discuss automated testing and its' role in Agile environments. VersionOne is recognized by agile practitioners as a leader in Agile project management tools and JoEllen is the lead tester on their agile development team. Automated Testing of Agile Projects vs. Waterfall Projects JoEllen had this to say about the difference: "Automation is a critical component to maintaining agility, and is a priority for the entire team through established practices/disciplines and a focus on continuous improvement. Continuous integration/builds, unit, functional & integration test execution and continuous/automated deployment are common examples of applying automation beyond the scope of traditional automated tests." "In contrast, waterfall projects will tend not to invest in automation as a priority, primarily due to the phased investment and handoff mindset that accompanies that approach. At best, builds may leverage some automation, and a phase of a waterfall project may be dedicated to functional test automation for long-term support/future release needs, but it's rare to find much more in a back-end-loaded waterfall project." Agile Tester Skills Regarding agile tester skills, JoEllen said that an Agile tester will have a more team oriented approach and use a wide variety of tools. "In addition to a quality/test-oriented outlook, an agile tester will incorporate a wide array of testing tools and technologies into their testing efforts. They are an integral member of the development team, participating fully in the development process, from design through delivery & deployment. A deep understanding of the domain and features under development is essential to developing tests that provide meaningful feedback. Testers may also work on several stories or features at one time, so tools to organize and juggle multiple testing activities are critical. They will have adapted to a more collaborative, team-based approach to the iterative/continuous delivery of high-quality, working software as a measure of progress." Automation Responsibilities on an Agile Team "The team! Literally anyone who does anything repetitively more than 2 or 3 times in a short period of time should and will automate it, preferably with a well known tool such as TestComplete, open source options or homegrown scripts/utilities. Testers generally shoulder the development of functional test automation, since that activity can parallel the actual feature development." Deciding what Tests get Automated in Agile Environments "The individuals doing detailed planning for a given user story (i.e. product owner(s), developers, testers) may make some of those decisions at that time. Developers and testers working on the story make detailed decisions about where automation can be used to trigger acceptance/signal completion of the story. During testing, testers may see opportunities for adhoc, on the fly automation to increase test coverage." Maintaining and Reviewing Automated Tests in Agile Environments "Since tests are often created by tester-developer and/or tester-product owner pairs, there's a built-in review process of the test scope and validation level as part of story acceptance. Otherwise, the collaborative planning to define the tests, then the collaborative acceptance with the product owner provides ample shared knowledge and review. The end result is robust, resilient automated tests which provide the backbone of regression testing, and any test failures from regression testing may also trigger a test review." "Ideally, the entire team maintains all tests, whether unit, functional, or any other. However, testers may use specialized tools for GUI-level tests that require their involvement to maintain tests over time. To reduce the possible silo-ing of this information, the tools should be carefully chosen to be accessible to all team members, information should be readily available, and the test inventory should be continuously culled to prevent test bloat. Beware of the tester building a mammoth suite of tests in a specialized environment as a job security strategy." Agile Teams Automate Black Box and White Box Tests "For functional/acceptance tests, it's most common for there to be a combination of white-box (under-the-covers) and UI automation (black-box) tests. The white-box tests tend to be less fragile, so are less costly to maintain, whereas the black-box UI-driven tests need to be leveraged judiciously due to the cost to develop and maintain the tests over time. Ideally, testers are involved in both, so that the black-box tests can supplement instead of duplicating white-box tests." Implementing Test Driven Development "By definition, TDD is a developer coding practice (via a unit test framework), not an agile tester practice. Simply, a developer will write a failing unit test, add the associated code to satisfy the test, then re-run those tests to ensure success. Subsequent refactoring/redesign of the resulting code is done whilst keeping the unit tests running 'Green'. The standard TDD mantra is Red-Green-Refactor." "However, the same concept (i.e. test-first) is definitely applied to higher-level functional and acceptance testing, whether automated or executed manually. Essentially, detailed requirements for a user story are expressed in the form of acceptance tests, which can then be captured in advance in a test automation environment such that they will eventually run 'green' once the functionality is in place, signaling completion (they may run 'yellow' on incomplete stories)." When Not to Use Automation in an Agile Environment It's important to know when and when not to automate tests, this holds true no matter what methodology is used. JoEllen said that if a test is only performed once then it shouldn't be automated. "One-time tasks and exploratory testing/edge cases shouldn't be automated. Edge cases are, by definition, usually one-off test cases, and the effort to automate edge cases generally does not pay off. Exploratory tests are best used to gain knowledge of a new feature and then tweak or revise tests based on the new knowledge." Switching from Waterfall to Agile When asked if she had any advice to give testers who are switching from a Waterfall project to an Agile project, JoEllen had this to say: "Be aware of the change in mindset/culture. Experiment with modern tools. Keep the big picture of the application in mind and consider individual user stories within the bigger context. Think holistically, and encourage the rest of the team to do so as well." Functional, GUI, Regression and Performance Testing in an Agile Environment JoEllen explained that all testing types have a place in an Agile shop, it is just a matter of how they are used. "Unit, functional, acceptance, GUI (some of which become regression), performance and load testing are all used. We try to select the most appropriate tool for the team and the type of test we're performing, so we have several different tools in our arsenal." "All of these testing types have a place in an agile environment. However, it's important to incorporate them in an Agile way, rather than a prescriptive way. Balance the quality goals of the team with the requirements and risk of individual user stories." Automated Testing Starts on Day One The automated testing process begins the same time as development in an Agile shop. JoEllen said, "The testing process begins at the same time as development - Day One." Automated Testing Tool Features - What is More Important to Agile Testers? "Most agile teams, regardless of any specific methodology, are looking for fast, reliable feedback from their automated tests. Fast, reliable feedback means timely results with minimal false failures, ideally incorporated as part of the continuous integration (build) pipeline." Automated Testing ROI in Agile Environments "It's really a prerequisite to becoming a truly agile team. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles. Thus, if ROI is producing high-quality software iteratively, then yes, you see a nice return on your automation." Automated Testing and Agile Methodologies JoEllen explained automated testing's role in different Agile methodologies: "In XP, automation is explicitly called out in a variety of disciplines (TDD, continuous integration, automated acceptance testing). Scrum teams may have incorporated those practices on their path to agility, but automation is not prescribed by the base methodology. Lean product development assumes automation in the pursuit of waste elimination." Implementing Automated Testing in Agile Environments - Challenges JoEllen explained the challenges frequently seen in Agile shops: "Availability of people & skills, a holistic team-oriented approach, striking the right balance between automation and excess test inventory which can result in drag. Some teams struggle with when to automate functional tests � whether to delay acceptance of a user story until the automated tests are complete, or to have automated test development as a separate activity. We've found that with the right tools, development of GUI-level tests can easily take place as standard work on a user story. However, automating functional tests on a story-by-story basis may lead to test bloat and may not provide the higher-level coverage that's desired from these tests. Teams may choose to write user stories to prioritize and schedule test development at that level." Automated Testing Benefits in an Agile Environment JoEllen said that the same benefits seen in other areas of testing are also seen in Agile shops. "Speed, quality, malleable code base (i.e. adaptable/open to change), feedback, efficiency are all benefits that we experience using automation for our tests." TestComplete in Agile Environments JoEllen had nice things to say about TestComplete for Agile testing: "A co-worker's prior company chose it over the high-end commercial offerings and really digs it for automating acceptance testing with both client-server and browser-based applications. It's both developer and tester-friendly, so it reduces the risk/tendency of specialization/single-role maintainability of the test suite." About VersionOne Since 2002, companies such as Adobe, BBC, Siemens, Disney, Dow Chemical, IBM, Lockheed Martin, Sony, 3M and Business Objects have turned to VersionOne to help provide greater value to their customers. Today more than 10,000 teams and 70,000 users in 50 countries use VersionOne's Agile project management tools to streamline and standardize their Agile development efforts - See more at: http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/#sthash.qR2YBNmZ.dpuf |
